1. Field of the Invention
This invention relates generally to computerized load testing and more particularly to a method and system for coordinating load testing of one or more sites under test over a network.
2. Description of the Related Art
Networks, such as the Internet, Intranets, and Extranets are fundamentally changing the way people and businesses communicate and transact with one another. The incredible growth of this new communications medium is well documented and many new applications are being invented at a phenomenal pace. The Internet is now used to provide informational Web-sites (e.g., drkoop.com(trademark)), portals/meta-sites (e.g., Yahoo!(trademark)), on-line auctions (e.g., ebay.com(trademark)), and collaborative communities (e.g., geocities.com(trademark)), and others. Traditional xe2x80x9cbrick and mortarxe2x80x9d businesses are moving on-line as well, providing such products and services as electronic commerce sites (e.g., amazon.com(trademark)), on-line stock trading (e.g., schwab.com(trademark)), and banking (e.g., citibank.com(trademark)). Ubiquitous corporate Intranets and Extranets utilizing Internet technologies provide similar applications and services to an internal or limited group of individuals and/or partners.
In a network environment, multiple users of client systems xe2x80x9cloadxe2x80x9d a host system by concurrently accessing the host system and placing various computing and communication demands on the host system. The ability to accept multiple concurrent accesses and accurately respond to the communication demands in a timely manner provide two key host system performance parameters. The dramatic success and continued growth of the Internet and the applications being built upon it have resulted in a substantial and increasing number of users accessing Web application host sites and, thus, an increase in the loads experienced by such sites. Thus, one of the key challenges facing Internet system hardware architects and application builders is how to design, build, and test a Web site to scale to meet ever growing user xe2x80x9cloadxe2x80x9d while maintaining satisfactory host system performance parameters.
The scalability of a host system, such as a Web application host site, is generally defined by two parameters: the maximum load achievable while maintaining: (1) host system reliability; and (2) acceptable host system response times. It can be quite straightforward to build a simple e-Commerce Web site to market or sell a product, but it is quite another to do so in a scalable, high-performance, reliable fashion. Several highly-publicized problems have been recently documented at leading Web sites. In fact, the Newport Group, a Massachusetts-based research company, currently estimates that 52 percent of all Web applications fail to scalexe2x80x94meaning they don""t perform acceptably under real-world usage increases.
Although host systems have been designed and implemented for many years, effectively, efficiently, and economically determining host system scalability has proved generally elusive. Several factors associated with Internet host systems contribute to making scalability an elusive, ever changing goal. First, the rapid growth trends and adoption of the Internet in general forces Internet host system builders to continually upgrade and reassess their ability to scale. A second factor stems from the challenging time-to-market pressures facing Web-centric companies, where predicting scalability can give way to the first and foremost goal of publicly activating the host systems. Third, new Web application technologies introduce increasing demands upon the technology infrastructure that companies have set up to provide their Internet presence. Broadband Internet connectivity, streaming audio/video, and voice-over-IP are examples of new Web application technologies which will make scaling Internet applications even more difficult. Fourth, technologies and requirements are advancing rapidly enough that human resources with adequate problem domain and solution knowledge for optimal Internet implementation and performance are scarce commodities in the current information technology workforce.
Load testing of Web applications can empirically determine host system scalability. However, effective, efficient, and economical Internet host system load testing has proved quite challenging. Traditionally, Web host application sites were load tested manually, which is neither a simple nor practical task. Using a manual approach to test an Internet host site under a load of 10,000 users would require hiring 10,000 people and giving them all a computer and network (intranet) or Internet access to the site to be tested. An additional drawback is that manual testing is non-repeatable in that the actions of large numbers of users are not predictable (repeatability being a hallmark of good testing practice).
In order to address some of the drawbacks of manual load testing, several contemporary vendors have developed load test applications that can simulate hundreds and even thousands of users accessing an Internet application. These tools commonly allow user behavior to be defined using some type of programmable script.
Many, if not all, load test tools are inadequate to address the needs of Internet site load testing. For instance, typical load testing tools are not optimized to test via the Internet and thus do not take in consideration the effects of the public nature of the Internet in performance. In addition, typical load test tools are limited (or at least untested) in their ability to truly simulate beyond a few thousand simultaneous users. Another drawback is that typical load testing tools can only effectively test one, independent target system at a time within an organization or a group of organizations. Therefore, typical conventional tools are unable to test multiple sites simultaneously.
In one embodiment, the present invention includes a method of load testing at least one site under test (SUT) in accordance with at least one load test request. The load test request provides information for load testing the at least one SUT. The load test request is reviewed, and a load test corresponding to the load test request is generated. The load test is allocated among a plurality of load cells. Load test data is sent, in accordance with the load test, from the plurality of load cells to the at least one SUT. Response data from the SUT is received by the load cells. At least one embodiment of the method further includes generating a load test report based on the response data. Another embodiment of the method provides for receiving and reviewing a plurality of load test requests pertaining to a SUT. A further embodiment of the method provides for receiving and reviewing a plurality of load test requests for each of a plurality of SUTs.
In another embodiment, the present invention includes a computer-readable medium having a computer program accessible therefrom, the computer program including instructions for performing the method of load testing the site(s) under test in accordance with the load test request(s). Note: [object] [s] means at least one [object]. The computer-readable medium includes any of the following: a magnetic storage medium, including disk and tape storage medium; an optical storage medium, including compact disk memory and a digital video disk storage medium; a nonvolatile memory storage memory; a volatile storage medium; and data transmission medium including packets of electronic data and electromagnetic waves modulated in accordance with the instructions.
Allocating the load test among the plurality of load cells includes, in at least one embodiment, determining whether a sufficient number of load cells are available to collectively conduct the load test request and, if so, distributing the load test among the available load cells until the load test is fully allocated. In at least one embodiment, the load cells to which the load test is allocated are geographically remote from the SUT.
In addition to the method discussed above, another embodiment of the invention includes a network system to load test at least one SUT in accordance with at least one load test request submitted by a user system. The network system includes at least one computer. The computer includes circuitry to receive a load test request from the user system. The computer also includes a memory and a processor that are electronically coupled to each other. In an alternative embodiment, the processor further includes circuitry to receive, via the Internet, a load test request from a user system.
The network system also includes program code. The program code is stored in the memory on the computer and is executable by the processor. The program code includes instructions for reviewing the load test request and for generating a load test based on the load test request. The program code also includes instructions for allocating the load test among a plurality of load cells and for sending load test data from the load cells to the at least one SUT. The program code also includes instructions for receiving responses from the SUT and for generating a load test report based upon the received responses. Another embodiment of the network system provides that the program code is logically partitioned into a plurality of functional subsystems. The functional subsystems include a public web site layer, a private web site layer, a load test database, a load test system manager, a work allocator, and a data collection system.
Another embodiment of the network system provides that the load test database (one of the functional subsystems mentioned above) is capable of storing login information, security information, captured user simulation scripts, and scenarios. A further embodiment of the load test database further provides that the load test database includes a relational database server.
A further embodiment of the network system provides that each load cell in the plurality of load cells includes a load cell manager, a load cell test executive, and a data buffer coupled to both the load cell manager and the load cell test executive.
Another embodiment of the invention includes an apparatus including a means for reviewing at least one load test request. The at least one load test request includes information for load testing at least one SUT. The apparatus also includes a means for generating a load test for each of the load test requests. The apparatus also includes a means for allocating the load tests to a plurality of load cells and a means for sending load test data from the load cells to the at least one SUT. The apparatus also includes a means for receiving responses from the SUT regarding the load test data.
Another embodiment of the present invention includes an apparatus for load testing at least one SUT in accordance with at least one load test request. The load test request is generated with a user data processing system. The apparatus includes a plurality of load cells capable of communicating with the SUT. The apparatus also includes a load test system. The load test system is capable of communicating with the plurality of load cells and with the user data processing system. The load test system includes a work allocator. The work allocator is capable of receiving the at least one load test request and is also capable of querying the plurality of load cells to select a group of load cells that is available to collectively process the load test request. The work allocator is also capable of selecting the group of load cells to collectively process the load test request.
Another embodiment of the invention includes a method of load testing a plurality of SUT""s, wherein the load is not necessarily allocated among a plurality of load cells. The load is allocated to at least one load cell. At least one embodiment provides for reviewing and processing at least one load test, where the load test request indicates load testing for a plurality of SUT""s. At least one embodiment further provides for reviewing and processing multiple load test requests for the plurality of SUT""s.
Another embodiment of the invention includes a system to load test a plurality of SUT""s, wherein the load is not necessarily allocated among a plurality of load cells. Another embodiment of the invention includes a computer-readable storage medium having a computer program with instructions for load testing a plurality of SUT""s. Another embodiment of the invention includes an apparatus capable of load testing a plurality of SUT""s, where the load is allocated to at least one load cell. Another embodiment of the invention includes another apparatus capable of load testing a plurality of SUT""s, where the load test request is generated with a user data processing system.
Another embodiment of the invention includes a method for load testing at least one SUT in accordance with a plurality of pending load test requests relating to the same SUT. The multiple pending load tests may provide for concurrent load testing of the SUT, or staggered testing. Another embodiment of the present invention includes a network system to load test at least one SUT in accordance with multiple pending load test requests. Another embodiment of the invention includes a computer-readable storage medium having a computer program with instructions for load testing at least one SUT in accordance with a plurality of concurrently-pending load test requests. Another embodiment of the invention includes an apparatus capable of load testing at least one SUT in accordance with a plurality of concurrently-pending load test requests. Another embodiment of the invention includes another apparatus capable of load testing at least one SUT in accordance with a plurality of load test requests, where the load test requests are generated with a user data processing system.